Skip to content

[SPARK-35031][PYTHON] Port Koalas operations on different frames tests into PySpark#32133

Closed
xinrong-meng wants to merge 5 commits into
apache:masterfrom
xinrong-meng:port.test_ops_on_diff_frames
Closed

[SPARK-35031][PYTHON] Port Koalas operations on different frames tests into PySpark#32133
xinrong-meng wants to merge 5 commits into
apache:masterfrom
xinrong-meng:port.test_ops_on_diff_frames

Conversation

@xinrong-meng

@xinrong-meng xinrong-meng commented Apr 12, 2021 •
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Now that we merged the Koalas main code into the PySpark code base (#32036), we should port the Koalas operations on different frames unit tests to PySpark.

Why are the changes needed?

Currently, the pandas-on-Spark modules are not tested fully. We should enable the operations on different frames unit tests.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Enable operations on different frames unit tests.

Keyword:SPARK-34849

@SparkQA

SparkQA commented Apr 12, 2021

Copy link
Copy Markdown

Test build #137240 has finished for PR 32133 at commit 34b8273.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@xinrong-meng xinrong-meng changed the title [WIP][SPARK-35031][PYTHON] Port Koalas operations on different DataFrames tests into PySpark [WIP][SPARK-35031][PYTHON] Port Koalas operations on different frames tests into PySpark Apr 12, 2021
@SparkQA

SparkQA commented Apr 12, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41820/

@SparkQA

SparkQA commented Apr 12, 2021

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41820/

ueshin
ueshin reviewed Apr 12, 2021
View reviewed changes
Comment thread dev/sparktestsupport/modules.py Outdated
@SparkQA

SparkQA commented Apr 12, 2021

Copy link
Copy Markdown

Test build #137244 has finished for PR 32133 at commit be499d5.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

ueshin
ueshin reviewed Apr 12, 2021
View reviewed changes
Comment thread python/pyspark/pandas/tests/test_ops_on_diff_frames.py Outdated
Comment thread python/pyspark/pandas/tests/test_ops_on_diff_frames.py Outdated
Comment thread python/pyspark/pandas/tests/test_ops_on_diff_frames_groupby.py Outdated
Comment thread python/pyspark/pandas/tests/test_ops_on_diff_frames_groupby.py Outdated
@SparkQA

SparkQA commented Apr 12, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41824/

@SparkQA

SparkQA commented Apr 12, 2021

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41824/

@SparkQA

SparkQA commented Apr 12, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41827/

@SparkQA

SparkQA commented Apr 12, 2021

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41827/

ueshin
ueshin approved these changes Apr 13, 2021
View reviewed changes

@ueshin ueshin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pending tests.

@SparkQA

SparkQA commented Apr 13, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41834/

@SparkQA

SparkQA commented Apr 13, 2021

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41834/

@SparkQA

SparkQA commented Apr 13, 2021

Copy link
Copy Markdown

Test build #137247 has finished for PR 32133 at commit c6c07a2.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@xinrong-meng xinrong-meng changed the title [WIP][SPARK-35031][PYTHON] Port Koalas operations on different frames tests into PySpark [SPARK-35031][PYTHON] Port Koalas operations on different frames tests into PySpark Apr 13, 2021
@xinrong-meng xinrong-meng marked this pull request as ready for review April 13, 2021 02:10
@xinrong-meng

xinrong-meng commented Apr 13, 2021

Copy link
Copy Markdown
Member Author

CC @HyukjinKwon @itholic

@HyukjinKwon

HyukjinKwon commented Apr 13, 2021

Copy link
Copy Markdown
Member

Merged to master.

@itholic

itholic commented Apr 13, 2021

Copy link
Copy Markdown
Contributor

Nice

@SparkQA

SparkQA commented Apr 13, 2021

Copy link
Copy Markdown

Test build #137255 has finished for PR 32133 at commit d0ea72a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ueshin ueshin ueshin approved these changes

Assignees

No one assigned

Labels

BUILD CORE PYTHON

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xinrong-meng @SparkQA @HyukjinKwon @itholic @ueshin